15 research outputs found
Smooth and Strong PCPs
Probabilistically checkable proofs (PCPs) can be verified based only on a constant amount of random queries, such that any correct claim has a proof that is always accepted, and incorrect claims are rejected with high probability (regardless of the given alleged proof). We consider two possible features of PCPs:
- A PCP is strong if it rejects an alleged proof of a correct claim with probability proportional to its distance from some correct proof of that claim.
- A PCP is smooth if each location in a proof is queried with equal probability.
We prove that all sets in NP have PCPs that are both smooth and strong, are of polynomial length, and can be verified based on a constant number of queries. This is achieved by following the proof of the PCP theorem of Arora, Lund, Motwani, Sudan and Szegedy (JACM, 1998), providing a stronger analysis of the Hadamard and Reed - Muller based PCPs and a refined PCP composition theorem. In fact, we show that any set in NP has a smooth strong canonical PCP of Proximity (PCPP), meaning that there is an efficiently computable bijection of NP witnesses to correct proofs. This improves on the recent construction of Dinur, Gur and Goldreich (ITCS, 2019) of PCPPs that are strong canonical but inherently non-smooth.
Our result implies the hardness of approximating the satisfiability of "stable" 3CNF formulae with bounded variable occurrence, where stable means that the number of clauses violated by an assignment is proportional to its distance from a satisfying assignment (in the relative Hamming metric). This proves a hypothesis used in the work of Friggstad, Khodamoradi and Salavatipour (SODA, 2019), suggesting a connection between the hardness of these instances and other stable optimization problems
Pseudointelligence: A Unifying Framework for Language Model Evaluation
With large language models surpassing human performance on an increasing
number of benchmarks, we must take a principled approach for targeted
evaluation of model capabilities. Inspired by pseudorandomness, we propose
pseudointelligence, which captures the maxim that "(perceived) intelligence
lies in the eye of the beholder". That is, that claims of intelligence are
meaningful only when their evaluator is taken into account. Concretely, we
propose a complexity-theoretic framework of model evaluation cast as a dynamic
interaction between a model and a learned evaluator. We demonstrate that this
framework can be used to reason about two case studies in language model
evaluation, as well as analyze existing evaluation methods.Comment: EMNLP 2023 Finding
Rigid Matrices From Rectangular PCPs
We introduce a variant of PCPs, that we refer to as rectangular PCPs, wherein
proofs are thought of as square matrices, and the random coins used by the
verifier can be partitioned into two disjoint sets, one determining the row of
each query and the other determining the column.
We construct PCPs that are efficient, short, smooth and (almost-)rectangular.
As a key application, we show that proofs for hard languages in ,
when viewed as matrices, are rigid infinitely often. This strengthens and
simplifies a recent result of Alman and Chen [FOCS, 2019] constructing explicit
rigid matrices in FNP. Namely, we prove the following theorem:
- There is a constant such that there is an FNP-machine
that, for infinitely many , on input outputs matrices
with entries in that are -far (in Hamming distance)
from matrices of rank at most .
Our construction of rectangular PCPs starts with an analysis of how
randomness yields queries in the Reed--Muller-based outer PCP of Ben-Sasson,
Goldreich, Harsha, Sudan and Vadhan [SICOMP, 2006; CCC, 2005]. We then show how
to preserve rectangularity under PCP composition and a smoothness-inducing
transformation. This warrants refined and stronger notions of rectangularity,
which we prove for the outer PCP and its transforms.Comment: 36 pages, 3 figure
A Theory of Unsupervised Translation Motivated by Understanding Animal Communication
Recent years have seen breakthroughs in neural language models that capture
nuances of language, culture, and knowledge. Neural networks are capable of
translating between languages -- in some cases even between two languages where
there is little or no access to parallel translations, in what is known as
Unsupervised Machine Translation (UMT). Given this progress, it is intriguing
to ask whether machine learning tools can ultimately enable understanding
animal communication, particularly that of highly intelligent animals. Our work
is motivated by an ambitious interdisciplinary initiative, Project CETI, which
is collecting a large corpus of sperm whale communications for machine
analysis.
We propose a theoretical framework for analyzing UMT when no parallel data
are available and when it cannot be assumed that the source and target corpora
address related subject domains or posses similar linguistic structure. The
framework requires access to a prior probability distribution that should
assign non-zero probability to possible translations. We instantiate our
framework with two models of language. Our analysis suggests that accuracy of
translation depends on the complexity of the source language and the amount of
``common ground'' between the source language and target prior.
We also prove upper bounds on the amount of data required from the source
language in the unsupervised setting as a function of the amount of data
required in a hypothetical supervised setting. Surprisingly, our bounds suggest
that the amount of source data required for unsupervised translation is
comparable to the supervised setting. For one of the language models which we
analyze we also prove a nearly matching lower bound.
Our analysis is purely information-theoretic and as such can inform how much
source data needs to be collected, but does not yield a computationally
efficient procedure
UniMASK: Unified Inference in Sequential Decision Problems
Randomly masking and predicting word tokens has been a successful approach in
pre-training language models for a variety of downstream tasks. In this work,
we observe that the same idea also applies naturally to sequential
decision-making, where many well-studied tasks like behavior cloning, offline
reinforcement learning, inverse dynamics, and waypoint conditioning correspond
to different sequence maskings over a sequence of states, actions, and returns.
We introduce the UniMASK framework, which provides a unified way to specify
models which can be trained on many different sequential decision-making tasks.
We show that a single UniMASK model is often capable of carrying out many tasks
with performance similar to or better than single-task models. Additionally,
after fine-tuning, our UniMASK models consistently outperform comparable
single-task models. Our code is publicly available at
https://github.com/micahcarroll/uniMASK.Comment: NeurIPS 2022 (Oral). A prior version was published at an ICML
Workshop, available at arXiv:2204.1332
The Corrective Feedback Instrument Revised
Results of an exploratory factor analysis study designed to examine and refine the Corrective Feedback Instrument are reported. The Corrective Feedback Instrument-Revised replaces the Corrective Feedback Instrument and consists of 30 items that can be used to encourage conversations about giving, receiving, and clarifying corrective feedback in group settings
Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformers
Randomly masking and predicting word tokens has been a successful approach in
pre-training language models for a variety of downstream tasks. In this work,
we observe that the same idea also applies naturally to sequential decision
making, where many well-studied tasks like behavior cloning, offline RL,
inverse dynamics, and waypoint conditioning correspond to different sequence
maskings over a sequence of states, actions, and returns. We introduce the
FlexiBiT framework, which provides a unified way to specify models which can be
trained on many different sequential decision making tasks. We show that a
single FlexiBiT model is simultaneously capable of carrying out many tasks with
performance similar to or better than specialized models. Additionally, we show
that performance can be further improved by fine-tuning our general model on
specific tasks of interest.Comment: Superseded by arXiv:2211.1086